Machine Learning Methods for Image Analysis in Medical Applications, from Alzheimer\u27s Disease, Brain Tumors, to Assisted Living

Healthcare has progressed greatly nowadays owing to technological advances, where machine learning plays an important role in processing and analyzing a large amount of medical data. This thesis investigates four healthcare-related issues (Alzheimer\u27s disease detection, glioma classification, human fall detection, and obstacle avoidance in prosthetic vision), where the underlying methodologies are associated with machine learning and computer vision. For Alzheimer’s disease (AD) diagnosis, apart from symptoms of patients, Magnetic Resonance Images (MRIs) also play an important role. Inspired by the success of deep learning, a new multi-stream multi-scale Convolutional Neural Network (CNN) architecture is proposed for AD detection from MRIs, where AD features are characterized in both the tissue level and the scale level for improved feature learning. Good classification performance is obtained for AD/NC (normal control) classification with test accuracy 94.74%. In glioma subtype classification, biopsies are usually needed for determining different molecular-based glioma subtypes. We investigate non-invasive glioma subtype prediction from MRIs by using deep learning. A 2D multi-stream CNN architecture is used to learn the features of gliomas from multi-modal MRIs, where the training dataset is enlarged with synthetic brain MRIs generated by pairwise Generative Adversarial Networks (GANs). Test accuracy 88.82% has been achieved for IDH mutation (a molecular-based subtype) prediction. A new deep semi-supervised learning method is also proposed to tackle the problem of missing molecular-related labels in training datasets for improving the performance of glioma classification. In other two applications, we also address video-based human fall detection by using co-saliency-enhanced Recurrent Convolutional Networks (RCNs), as well as obstacle avoidance in prosthetic vision by characterizing obstacle-related video features using a Spiking Neural Network (SNN). These investigations can benefit future research, where artificial intelligence/deep learning may open a new way for real medical applications

A spiking neural network model for obstacle avoidance in simulated prosthetic vision

Limited by visual percepts elicited by existing visual prosthesis, it’s necessary to enhance its functionality to fulfill some challenging tasks for the blind such as obstacle avoidance. This paper argues that spiking neural networks (SNN) are effective techniques for object recognition and introduces for the first time a SNN model for obstacle recognition to as- sist blind people wearing prosthetic vision devices by modelling and classifying spatio- temporal (ST) video data. The proposed methodology is based on a novel spiking neural network architecture, called NeuCube as a general framework for video data modelling in simulated prosthetic vision. As an integrated environment including spiking trains en- coding, input variable mapping, unsupervised reservoir training and supervised classifier training, the NeuCube consists of a spiking neural network reservoir (SNNr) and a dy- namic evolving spiking neural network classifier (deSNN). First, input data is captured by visual prosthesis, then ST feature extraction is utilized in the low-resolution prosthetic vi- sion generated by prostheses. Finally such ST features are fed to the NeuCube to output classification result of obstacle analysis for an early warning system to be activated. Ex- periments on collected video data and comparison with other computational intelligence methods indicate promising results. This makes it possible to directly utilize available neu- romorphic hardware chips, embedded in visual prostheses, to enhance significantly their functionality. The proposed NeuCube-based obstacle avoidance methodology provides use- ful guidance to the blind, thus offering a significant improvement of current prostheses and potentially benefiting future prosthesis wearers

Enlarged Training Dataset by Pairwise GANs for Molecular-Based Brain Tumor Classification

This paper addresses issues of brain tumor subtype classification using Magnetic Resonance Images (MRIs) from different scanner modalities like T1 weighted, T1 weighted with contrast-enhanced, T2 weighted and FLAIR images. Currently most available glioma datasets are relatively moderate in size,and often accompanied with incomplete MRIs in different modalities. To tackle the commonly encountered problems of insufficiently large brain tumor datasets and incomplete modality of image for deep learning, we propose to add augmented brain MR images to enlarge the training dataset by employing a pairwise Generative Adversarial Network (GAN) model. The pairwise GAN is able to generate synthetic MRIs across different modalities. To achieve the patient-level diagnostic result, we propose a post-processing strategy to combine the slice-level glioma subtype classification results by majority voting. A two-stage course-to-fine training strategy is proposed to learn the glioma feature using GAN-augmented MRIs followed by real MRIs. To evaluate the effectiveness of the proposed scheme, experiments have been conducted on a brain tumor dataset for classifying glioma molecular subtypes: isocitrate dehydrogenase 1 (IDH1) mutation and IDH1 wild-type. Our results on the dataset have shown good performance (with test accuracy 88.82%). Comparisons with several state-of-the-art methods are also included

Deep semi-supervised learning for brain tumor classification

Background: This paper addresses issues of brain tumor, glioma, classification from four modalities of Magnetic Resonance Image (MRI) scans (i.e., T1 weighted MRI, T1 weighted MRI with contrast-enhanced, T2 weighted MRI and FLAIR). Currently, many available glioma datasets often contain some unlabeled brain scans, and many datasets are moderate in size. Methods: We propose to exploit deep semi-supervised learning to make full use of the unlabeled data. Deep CNN features were incorporated into a new graph-based semi-supervised learning framework for learning the labels of the unlabeled data, where a new 3D-2D consistent constraint is added to make consistent classifications for the 2D slices from the same 3D brain scan. A deep-learning classifier is then trained to classify different glioma types using both labeled and unlabeled data with estimated labels. To alleviate the overfitting caused by moderate-size datasets, synthetic MRIs generated by Generative Adversarial Networks (GANs) are added in the training of CNNs. Results: The proposed scheme has been tested on two glioma datasets, TCGA dataset for IDH-mutation prediction (molecular-based glioma subtype classification) and MICCAI dataset for glioma grading. Our results have shown good performance (with test accuracies 86.53% on TCGA dataset and 90.70% on MICCAI dataset). Conclusions: The proposed scheme is effective for glioma IDH-mutation prediction and glioma grading, and its performance is comparable to the state-of-the-art

Human Fall Detection using Co-Saliency-Enhanced Deep Recurrent Convolutional Neural Networks

This paper addresses issues of fall detection from videos for e-healthcare and assisted-living. Instead of using hand-crafted features from videos, we exploit a dedicated recurrent convolutional network (RCN) architecture for fall detection in combination with co-saliency enhancement. In the proposed scheme, the recurrent neural network (RNN) is realized by Long Short-Term Memory (LSTM) connecting to a set of Convolutional Neural Networks (CNNs), where each video is modelled as an ordered sequence, containing several frames. In such a way, the sequential information in video is preserved. To further enhance the performance, we propose to employ co-saliency-enhanced video frames as the inputs of RCN, where salient human activity regions are enhanced. Experimental results have shown that the proposed scheme is effective. Further, our results have shown very good test performance (accuracy 98.12%), and employing the co-saliency-enhanced RCN has led to the improvement in performance (0.70% on test) as comparing to that without co-saliency. Comparisons with two existing methods have provided further support to effectiveness of the proposed scheme

Human fall detection using segment-level CNN features and sparse dictionary learning

This paper addresses issues in human fall detection from videos. Unlike using handcrafted features in the conventional machine learning, we extract features from Convolutional Neural Networks (CNNs) for human fall detection. Similar to many existing work using two stream inputs, we use a spatial CNN stream with raw image difference and a temporal CNN stream with optical flow as the inputs of CNN. Different from conventional two stream action recognition work, we exploit sparse representation with residual-based pooling on the CNN extracted features, for obtaining more discriminative feature codes. For characterizing the sequential information in video activity, we use the code vector from long-range dynamic feature representation by concatenating codes in segment-levels as the input to a SVM classifier. Experiments have been conducted on two public video databases for fall detection. Comparisons with six existing methods show the effectiveness of the proposed method

Co-Saliency-Enhanced Deep Recurrent Convolutional Networks for Human Fall Detection in E-Healthcare

This paper addresses the issue of fall detection from videos for e-healthcare and assisted-living. Instead of using conventional hand-crafted features from videos, we propose a fall detection scheme based on co-saliency-enhanced recurrent convolutional network (RCN) architecture for fall detection from videos. In the proposed scheme, a deep learning method RCN is realized by a set of Convolutional Neural Networks (CNNs) in segment-levels followed by a Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), to handle the time-dependent video frames. The co-saliency-based method enhances salient human activity regions hence further improves the deep learning performance. The main contributions of the paper include: (a) propose a recurrent convolutional network (RCN) architecture that is dedicated to the tasks of human fall detection in videos; (b) integrate a co-saliency enhancement to the deep learning scheme for further improving the deep learning performance; (c) extensive empirical tests for performance analysis and evaluation under different network settings and data partitioning. Experiments using the proposed scheme were conducted on an open dataset containing multicamera videos from different view angles, results have shown very good performance (test accuracy 98.96%). Comparisons with two existing methods have provided further support to the proposed scheme

3D Multi-Scale Convolutional Networks for Glioma Grading Using MR Images

This paper addresses issues of grading brain tumor, glioma, from Magnetic Resonance Images (MRIs). Although feature pyramid is shown to be useful to extract multi-scale features for object recognition, it is rarely explored in MRI images for glioma classification/grading. For glioma grading, existing deep learning methods often use convolutional neural networks (CNNs) to extract single-scale features without considering that the scales of brain tumor features vary depending on structure/shape, size, tissue smoothness, and locations. In this paper, we propose to incorporate the multi-scale feature learning into a deep convolutional network architecture, which extracts multi-scale semantic as well as fine features for glioma tumor grading. The main contributions of the paper are: (a) propose a novel 3D multi-scale convolutional network architecture for the dedicated task of glioma grading; (b) propose a novel feature fusion scheme that further refines multi-scale features generated from multi-scale convolutional layers; (c) propose a saliency-aware strategy to enhance tumor regions of MRIs. Experiments were conducted on an open dataset for classifying high/low grade gliomas. Performance on the test set using the proposed scheme has shown good results (with accuracy of 89.47%)

Co-saliency detection via inter and intra saliency propagation

The goal of salient object detection from an image is to extract the regions which capture the attention of the human visual system more than other regions of the image. In this paper a novel method is presented for detecting salient objects from a set of images, known as co-saliency detection. We treat co-saliency detection as a two-stage saliency propagation problem. The first inter-saliency propagation stage utilizes the similarity between a pair of images to discover common properties of the images with the help of a single image saliency map. With the pairwise co-salient foreground cue maps obtained, the second intra-saliency propagation stage refines pairwise saliency detection using a graph-based method combining both foreground and background cues. A new fusion strategy is then used to obtain the co-saliency detection results. Finally an integrated multi-scale scheme is employed to obtain pixel-level co-saliency maps. The proposed method makes use of existing saliency detection models for co-saliency detection and is not overly sensitive to the initial saliency model selected. Extensive experiments on three benchmark databases show the superiority of the proposed co-saliency model against the state-of-the-art methods both subjectively and objectively

Deep Learning and Multi-Sensor Fusion for Glioma Classification Using Multistream 2D Convolutional Networks

This paper addresses issues of brain tumor, glioma, grading from multi-sensor images. Different types of scanners (or sensors) like enhanced T1-MRI, T2-MRI and FLAIR, show different contrast and are sensitive to different brain tissues and fluid regions. Most existing works use 3D brain images from single sensor. In this paper, we propose a novel multistream deep Convolutional Neural Network (CNN) architecture that extracts and fuses the features from multiple sensors for glioma tumor grading/subcategory grading. The main contributions of the paper are: (a) propose a novel multistream deep CNN architecture for glioma grading; (b) apply sensor fusion from T1-MRI, T2-MRI and/or FLAIR for enhancing performance through feature aggregation; (c) mitigate overfitting by using 2D brain image slices in combination with 2D image augmentation. Two datasets were used for our experiments, one for classifying low/high grade gliomas, another for classifying glioma with/without 1p19q codeletion. Experiments using the proposed scheme have shown good results (with test accuracy of 90.87% for former case, and 89.39 % for the latter case). Comparisons with several existing methods have provided further support to the proposed scheme